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ABSTRACT 

The paper is a methodological inquiry into the interpretation of qualitative data. It explores 
a grounded theory approach to the synthesis of data, and examines, in particular, construction of 
categories. It focuses on ways of organizing data and attaching meaning, as research problems 
embedded In cultural context are explored. A qualitative research tr^ ning task, with 4-5 subjects 
per group (4 or 5 groups per class) evaluating comic strip culture, was used. We wanted to know 
how different ways of categorizing data lead to different interpretations of comic strip culture. In 
this regard we were looking at a) consistency within groups, and b) how groups differ. 
Implications of this study center around the idiosyncratic nature of qualitative research, issues 
related to generalizabili^. and relationships between training and non-training of researchers to 
the interpretation of data. The results of this Investigation reflect the systematic replication of 
three Independent studies. 

Perspective 

Qualitative research has gained in popularity but with few people well- 
trained in the methodology applying to its use. With the 'ncrease in usage there 
is also an increase in the need to understand some of the concerns innate in 
the procedures used when conducting qualitative investigations. It is 
recognized by almost all authorities (Eisner & Peshkin, 1990; Guba & Lincoln, 
1982; Howe & Eisenhart, 1990; LeCompte & Goetz. 1982) that qualitative 
research has a high subjective component in its interpretation and that this has 
been a major criticism leveled at this particular approach. The current research 
project, addressing this concern, has attempted to peel back the "veneers of 
phenomenological representation" (Constas, 1992, p. 254) to examine the 
process of category development which is often only implicitly recognized. 



iThis paper was presented at the Annual Conference of the Eastern 
Educational Research Association, Sarasota, Florida, February 9-12, 1994. 



2 



The rebuttal to the criticism of subjectivity in qualitative research tends to 
rest on 1 ) strengths of the methodology which compensate for the subjectivity 
inherent to some extent in the method, and 2) the assumption that training in 
coding methodology and other aspects of qualitative procedures will decrease 
these concerns. Based upon the second argument above, the investigators 
designed a methodological inquiry examining the elicitation of categories as 
part of a qualitative interpretation of a cartoon culture. Investigators were 
particularly interested in the effects of coding training on the elicitation of 
categories. 

As a methodological inquiry into the interpretation of qualitative data, 
these studies explore a grounded theory approach to the synthesis of data, i.e., 
generation of theory through the discovery of categories elicited from data 
(Glaser & Strauss, 1967). We are looking at the identification and building of 
categories as one part of the coding, categorizing, and thematic development 
sequence. Looking at research problems as embedded in cultural context, our 
concern here is in the organization and attachment of meaning to data. 

This research addresses some key components of an ongoing debate 
over the value and place of qualitative approaches to the understanding of 
problems in education and related fields. We see ourselves as dealing with 
"questions concerning the credibility and status of qualitative inquiry ... (and) the 
privatization of qualitative analysis" (Constas, 1992, p. 265). Eisner and 
Peshkin (1990) stress the need to explore and further develop models of 
qualitative research acceptable to the research community, and to which 
educational researchers might turn for direction. Newman and Benz (1992) ask 
that educational researchers reject the qualitative-quantitative dichotomy and 
think in terms of a synthesis of the two approaches. These perspectives involve, 
among other things, renewed examination of both research approaches in 
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terms of a broader paradigm or frame of reference than has often been used in 
the past. 

These studies focus on the data analysis phase of ethnographic 
research. The importance of the investigation is related to the tremendous 
complexity and layering of ethnographic interpretation. It is Geertz who 
observed ". . . that what we call our data are really our own constructions of 
other people's constructions of what they and their compatriots are up to . . . 
(and that it) is obscured because most of what we need to comprehend a 
particular event, ritual, custom, idea, or whatever is insinuated as background 
information before the thing itself is directly examined (1973, p. 9)." 

This particular research focuses on qualitative inquiry in relation to 
issues of reliability, validity, and generalization-concerns often raised by 
educational researchers, especially by "quantitative" researchers (Kirk & Miller, 
1S86). Addressing its inappropriateness for generalization, the value of 
qualitative research is often described as in the depth and richness of 
description it provides in particular and idiosyncratic cases, and not in 
generalization to other cases (Erickson, 1988; Peshkin, 1993). However, there 
are some recent claims by qualitative researchers such as Polkinghorne (1991) 
and Firestone (1993) that qualitative research can be generalized, in some 
ways, beyond the specific case. By looking at the elicitation of categories as 
part of the process of qualitative analysis, this research focuses on the above 
concerns. 
Objectives 

This is an investigation of the qualitative interpretation of a synthetic 
cartoon culture, controlling for methodological concerns. The different stages in 
the investigation will be referred to as Study 1 (Newman & MacDonald, 1993), 
Study 2 (MacDonald, Newman, Waite, & Potts. 1993), and Study 3 (which is not 
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reported anywhere except here). The investigation described here is a 
synthesis of the three studies, each study having been built upon an earlier one 
for purposes of controlling for coding effects and content differences. The 
investigation also has the strength of replication, since parts of each of the three 
studies were virtually identical, allowing us to estimate the effects of sample 
differences. 

Specifically, the purpose of the investigation was to examine some 
methodological aspects of qualitative data analysis. The objectives of the 
investigation are as follows: 

1) To replicate findings of previous studies 

2) To estimate the effect of training subjects in coding techniques on 
change in their interpretation 

3) To estimate the effect of training subjects in coding techniques towards 

increasing group consensus (within-group agreement) 

4) To estimte the effect of training subjects in coding techniques towards 

increasing generalizability across groups (across-group agreement) 

5) To estimate agreement between subjects for the identified categories 

6) To estimate the effect of content differences within and across groups 
Method and Data Source 

A comic strip culture was used as a common data set for studying the 
elicitation of categories in qualitative analysis by 1) individuals given the same 
data, and 2) groups consisting of the individuals who had already processed 
the data individually. Comparisons were made of 1) individual responses , 
2) intra-group responses, and 3) inter-group responses. A qualitative research 
training task, with 4-5 member groups evaluating comic strip culture, was used. 
We wanted to know how different ways of categorizing data lead to different 
interpretations of comic strip culture. In this regard we were looking at 
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percentages of agreement, i.e., a) consistency within groups, and b) how 
groups differ. 

We started with four graduate students who served as interviewees after 
reading comic strip material to be used with a graduate class later. Questions 
and approach to the cartoon culture simulation were modified on the basis of 
what we learned. In Week 9 of the semester, twenty-seven students in a 
graduate course focusing on education in cultural context, participated in a 
qualitative inquiry into the nature of culture and its analysis through a simulation 
using ethnographic techniques. In this initial inquiny. Study 1 (Newman and 
MacDonald, 1993), students doing the analysis had not been trained in 
ethnographic techniques. They had, however, been exposed over several 
sessions to characteristics of culture, and a range of concepts associated with 
culture and sub-cultures. In Study 2 (MacDonald, Newman, Waite, and Potts) 
and Study 3 students received some training in qualitative data gathering and 
analysis (see pages 6 and 12). 

The class activity was titled MAKING MEANING: AN EXPLORATION OF 
COMIC STRIP CULTURE , and resulted, in each of the three studies, in 27, 23, 
and 22 individual responses respectively, and 4-5 group responses, to the 
following set of questions: 1) Who are the main characters (in this culture)? 
2) Which characters have the most prestige (in this culture)? Why do you think 
this? *3) Describe this culture. What are the cultural values? 4) What general 
reaction and/or miscellaneous observations can you note? 5) How familiar are 
you with this comic strip? 6) How long did it take you to read the book? This 
paper focuses on students' responses to Question #3. 

The task consisted of students individually reading the comic strip book 
and writing their responses to the above questions outside class. Aftenwards, 
during one class session, approximately an hour and a half, the individual 
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responses were discussed in groups. Each group was asked to reach 
consensus on a cultural description of this comic strip, following the set of 
questions they had worked through individually. They were asked to put their 
group response in writing. 

Grouping, for purposes of this in-class activity, was by self-selection. In 
order to make the task more fun, students were assigned to groups of 4-5, on a 
first-come, first-serve basis, by raising their hands as they agreed with 
particular, casual statemer s (in the second and third studies, students 
"numbered off"). Class discussion followed the group process, including inquiry 
into change in their thinking as a result of attempting group consensus, i.e., 
group description of the culture. 

The task for Study 1 (Newman & MacDonald, 1993), conducted Spring 
1993, remained the same for Study 2 (MacDonald et al., 1993), conducted 
Summer 1993, and for Study 3, Fall 1993, making Studies 2 and 3 a replication 
of Study 1 , with the addition of some training in coding techniques. Basically, 
the training in Studies 2 and 3 involved repeated viewings of a video of a 
culture which was unfamiliar to students in the class; it involved instructing and 
coaching them in observation, note-taking, and concept development and 
crtegorization (Spradley, 1979). 

Study 2 is in two parts. Part I being equivalent to the first study where 
students were asked simply to describe the culture in the first third of the book, 
without reliance on any specific training. In Part II they were asked to apply their 
coding training to interpretation of the culture as depicted in either the second or 
third section of the book. 

Study 3 also had two parts. Part I of Study 3 was a replication of Study 2. 
In conducting these studies, students were assigned various portions of the text 
for different parts of the assignment. Study 1 ; Study 2, Part I; and Study 3, Part I 
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are all based on the first third of the text. Study 2, Part II and Study 3, Part II are 
based on the remaining two-thirds of the text. In Study 3 the students arrived at 
separate group consensus for the first and second part of the assignment. A 
split in group consensus along content lines (different parts of the text-first, 
second, third-had different content) allowed for creation of the second part of 
the study, which will compare effects of coding training with consensus arrived 
at in the latter part of the text. We are comparing coding differences based on 
overall consensus with coding differences based on content-specific 
differences. Part II of Study 3, then, used the same procedures as in the 
previous part, but also examined content differences. 
Analysis 

Students' written responses to Question #3 (above) were categorized 
and tallied in three ways. First, individual answers were written before the 
group session in class. We used those individual responses to develop 
categories describing the culture. Secondly, we tallied the number of 
individuals who identified similar categories. Third, we had the group respond- 
come to some consensus^-and tallied the group responses by categories 
generated. Then we compared the number of categories generated and the 
agreement of individuals compared to group agreement with the categories. 
In each of the three ways of framing the data above, we have looked at 
differences due to coding effects and content vcriation. The results from 
Study 1 are reported in Table 1. The results from Study 2, Parts I and II, are 
shown in Tables 2 and 3 respectively. Results from Study 3, Parts I and II, are 
reported in Tables 4 and 5 respectively. 



2We have operationally defined consensus to mean whatever the group 
agreed upon. 
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Results of Study 1 (Without training) 

Findings for Study 1 were: 1) Intra-group differences and perceptions 
were pronounced. Out of 19 possible categories, tliere was no group in whicli 
everyone selected any one category, although in some cases attention 
clustered in certain categories, 2) Group consensus varied considerably from 
individual reports, 3) Individuals reported more diverse answers than group 
consensus indicated, in spite of the fact that most individuals reported no 
change in atttitudes as a result of the experience, 4) Group consensus provided 
responses which were more simplified, and were focused on fewer categories 
than those of individuals, and 5) Unexpectedly, inter-group comparisons based 
on consensus yielded little agreement. 
Results of Study 2 (With training) 

Study 2 produced slightly different categories than Study 1 (See 
Tables 1 , 2, and 3). For the class in Study 2, Part II (the part that used their 
training in developing the categories), more categories (29 versus 34), and 
more values within categories, were identified, implying that training may have 
increased students' awareness and ability to discern increased numbers of 
categories and variations within categories. 

As one can see from Table 2, Study 2, Part I, this class, when given the 
same task as those in Study 1 , produced results which were not any more 
consistent than those in Study 1 ; there was still a wide range of differences in 
categories within and between groups. For instance, in Study 2, Part I, Group 3 
elicited only 13 out of the possible total of 29 categories generated by the class, 
whereas Group 4 came up with 23 of the 29 categories. Looking at the 
consenjjus items for Study 2, Part I, i.e., how many categories individuals 
identified compared with categories that the group agreed upon as existing, we 
found the following percentages of agreement for the five groups: 35%, 22%, 
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54%, 30%, and 28%. The averages of the percentages were about the same as 
in Study 1 (19%, 33%, and 50%)3. 

Table 3 summarizes the data that was collected from students in Study 2, 
Part II, where they were asked to apply their training in coding to the 
interpretation of the cartoon culture. As one can see from. Table 3, there were 
more categories produced here than in Study 1 or in Part I of Study 2. 
However, the percentages of agreement between the individual responses in 
the group and the consentu'; items for the group were less than those in 
Study 1 or in Part I of Study 2. The percentages for Study 2, Part II were: 29%, 
25%, b3%, 30%, and 26% (see Table 3). 

Students were asked if they had changed their mind on categories which 
they had generated individually as compared with the group's report of 
consensus items. Interestingly, a large majority of students (77% in Study 2) 
said the process of reaching group consensus did not change their own original 
opinion, and in addition, verbal reports from a majority of students indicated that 
the group consensus was reflective of their own opinion. These findings are 
contrary to the individual data indicating that there were many more individual 
differences than the group reported as consensus. This paradox was found in 
Study 1 ; Study 2, Part I; and Study 2, Part ll-with training as well as in the 
absence of training. 

It is interesting to note that in many cases where students reported no 
change in their views as a result of the group process, they did comment that 
they became aware of additional ideas and categories that they had not 
generated on their own. Although these comments indicated that more 

3There were originally four groups in Study 1, but one of the groups. Group 2, 
was eliminated since students in this group did not follow directions relative to 
consensus items. They recorded, "The values are the same as those of the 
culture-at-large." They did not list what the values were. 
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categories may have been discussed in the group process, what came out of 
the group were fewer categories-fewer than the individuals generated 
collectively, and, for the most part, fewer than individuals had generated prior to 
the group experience. 
Results of Study 3 (With tmining) 

Results of Study 3 replicate the findings of Study 1 and Study 2, Part I. In 
each case there was little agreement between individuals and group 
consensus, yet individuals reported high agreement between their response 
and group consensus. There was virtually no difference in Study 3 related to 
content. 

We were asked if the findings above would hold up for different content. 
It was pointed out that the book we used had different political and satirical 
content in the different sections-first, second, third. We found that Study 3 
replicated Studies 1 and 2 for each content area. There was no difference by 
content areas of the book. This, however, could be different with another kind of 
book, e.g., using a non-comic strip approach. 
Results & Conclusions 

Results are reported here as they match the six objectives of the 
investigation (see page 3). The first objective was to replicate findings of 
previous studies. As one can see from Tables 1 , 2, and 4 as well as the text 
above which describes our analysis, ..^e findings of Study 1 were replicated in 
Study 2. Part I and Study 3, Part I. 

The second objective was to estimate the effect of training subjects in 
coding techniques on change in their interpretation. This is where we found 
one of the most intriguing aspects of the whole investigation. We discovered 
that our subjects phenomenologically indicated high agreement with the group, 
while their responses indicated low agreement. There were no subjects who 
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indicated that there was high disagreement, between the individual responses 
and the group consensus. This was determined by a question that all subjects 
were asked, "Did your views change as a result of the group's attempt to reach 
consensus"? 

The third objective was to estimate the effect of training subjects in 
coding techniques towards increasing group consensus (within-group 
agreement). The findings in the differences in consensus between Study 2, 
Part I c\nd Study 2, Part II, and between Study I and Study 2, Part II. indicated 
that training had very little, if any, effect on within-group consensus. 

The fourth objective was to estimate the effect of training subjects in 
coding techniques towards increasing generalizability across groups (across- 
group agreement). As one can see from the final column of Tables 1-5, and the 
mean agreement for the separate studies between groups, there was relatively 
little agreement between groups for each of the categories. The average 
number of agreements between groups without training compared to the 
average number of groups with training indicated that training did not increase 
the amount of agreement between groups. There were two notable exceptions 
as seen in 1) Table 4--Study 3, Part I, Category 6, Importance of Relationships,, 
with 80% agreement, and 2) Table 5--Study 3, Part II, Category 9, Gender 
Differences, also with 80 /o agreement. The percentages of between-group 
agreement were similar regardless of training/no-training or content. This 
surprised us. We expected to have greater between-group agreement than 
within-group. We thought that the interaction between individuals necessary to 
achieve consensus would have a collapsing effect which would contribute to 
greater agreement between groups. In this investigation we did not find this to 
be true. 
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The fifth objective was to estimate agreement between subjects for the 
identified categories. Tables 1-5 indicated that the agreement between 
subjects on the elicitation of categories tended to increase when training was 
used. These mean increases were concomitant with increases in standard 
deviations and were not significantly different. They are as follows; 

Mean SD 

Study 2, Part I (No training) 4.97 2.78 
Part II (With training) 5.85 4.36 

Study 3 Part I (No training) 3.63 2.79 

Part II (With training) 4.29 3.24 

The sixth objective was to estimate the effect of content differences within 
and across groups. Study 3, Tables 4 and 5, reflect the findings that indicated 
content had virtually no effect on the respondents' percentage of agreement 
individually or in groups. 
Di scussion & Implications 

in Study 2 (MacDonald et al., 1993) and in Study 3, in part, we replicated 
Study 1 (Newman & MacDonald, 1993), but also looked at the effects of 
training students in coding methodology on interpretation of qualitative data and 
consensus within and between groups. Results of the second and third studies 
are very similar to the first study. Although it appears that training in coding 
methods increased the perceived number of categories, it seemed to have little 
or no effect on agreement within and between groups when compared to the 
first study, where students had not received training. There are a variety of 
possible explanations for this outcome. 

There were different individuals classifying student categories in Study 1 
as compared with Studies 2 and 3. This could explain some differences 
between Study 1 and Studies 2 and 3. However, it would not account for 
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difference or lack of difference in Parts I and il of Study 2 and parts I and II of 
Study 3; the person classifying the categories was the same for these. 

There were some overall differences in the makeup of the two classes, 
those in Study 2 being generally older than Study 1, more frequently in practice 
as teachers, administrators, counselors, or social workers; and, those in Study 1 
tending to be younger, non-practitioner, and in many cases, preparing for 
careers in counseling psychology. The class composition in Study 3 tended to 
be like a combination of the Study 1 and Study 2 groups, with the addition of a 
sub-set of students studying higher education. We cannot, at this point, gauge 
the impact that these differences had on outcomes of the three studies, but 
again, this would not have affected comparisons between Parts I and II of 
Study 2, and similarly, with Study 3. 

In general, there was a fair amount of naivete in the students asked to do 
the coding, which was reflected in how they did the coding and the consequent 
generation of categories. This naivete was obvious in looking at their attempted 
coding of the cartoon culture. This feature may or may not be more problematic 
in these studies than one would suspect to be the case for those who actually 
do qualitative research. 

As one can see from our data, individuals thought they actually received 
more information by discussing in group, yet what always came out of the group 
was a sub-set of ail the individual responses. It appears that one does get 
more information by discussing things in groups, but that the aggregated 
consensus in the group is less than the total of all the individual responses, and 
that agreement between groups on the consensus items was not any better 
than agreement between consensus items for individuals in the groups. 
Actually, the areas where the individuals had the highest agreement are not 
reflected in the consensus statement. 
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The person who analyzed the student categories for Studies 2 and 3 
reported difficulty and frustration since some of the students' categories and the 
way they elaborated on them depended upon so many nuances and subtleties. 
The use of a cartoon culture, especially a satirical one, may have presented 
some unique problems in this regard. For example, "sexism" in the cartoon 
culture was found in a sequence (there were other examples of "sexism" in the 
book as well) in which some of the book's male characters, in the presence of a 
very young female looking puzzled and dismayed, were describing one of the 
'ladies of tele-journalism,' Diane Sawyer. They said she was 'lookin' hotter 'n a 
tamale,' had lips which were 'plump and pink as a June cranberry,' has been 
'lookin' a might meaty in the drumsticks,' etc. (Breathed, 1992). 

In interpreting a passage such as the one about Diane Sawyer, we think 
that some students respond to the culture in a concrete way; i.e., they interpret 
"sexist" references as reflecting the meaning of the culture. Others, realizing it is 
satirical, appear to respond to the satire; i.e., picking up on very sexist 
comments, they say that the cartoon is arguing against sexism by making it 
sound stupid. Others seem to deal with it in terms of a counter-culture and its 
political implications. So the same event, depending on the students' 
perspectives, may be coded differently. 

One might argue that such a range of interpretation is a problem. We are 
taking the position that virtually al! qualitative researchers have a perspective, 
whether aware or not, and that these different perspectives are likely to produce 
different coding responses. Using a cartoon culture may only make these 
discrepancies more obvious. This experience with the cartoon culture m ght be 
a more accurate simulation of the real problems encountered in the 
interpretation of qualitative data than we originally realized. 
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Last, we do not know how much of these results we can attribute to the 
quality or extent of training received by students in this graduate class, since the 
time allotted for training was minimal-approximately an hour out of each of 
three class sessions. 
Importance of the Study 

This data support a variety of conjectures. First, if we have individuals 
interpreting culture, we could easily get a perception that is difficult to agree 
upon because it is based on idiosyncratic analysis of data describing cultural 
values. 

Secondly, recent mainstream thinking of qualitative and quantitative 
researchers suggests that (a) individuals are more likely to give rich, in-depth 
perceptions, (b) however, that individual data is less likely to be generalizable 
to other individuals perceiving this data. We originally thought that group data 
might be somewhat more generalizable. We thought we might use aggregated 
data with the expectation of more consistency and reliability, but perhaps less 
rich with idiosyncratic differences. However, our data suggest that 
generalizability was not increased by the aggregated group consensus data. 
This could be due to the coding problems we have identified previously, or to an 
inherent aspect of this type of research, as we discussed earlier. 

Third, the group produced less in the way of in-depth perceptions, in 
these studies, fewer and more general categories than did individuals. 

Fourth, the results are contrary to recent attempts by some qualitative 
researchers, such as Polkinghorne (1991), to generalize with qualitative 
research, although they say that it is a different type of generalization. 

Last, one may wish to train observers to increase inter-rater reliability; 
however, trained observers might acquire a pre-set and see what they were 
trained to see, or what they were trained to see as important. It is difficult to 
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understand how one can be trained to obsen^/e independently of their own 
personal experiences and values. Training people might increase the reliability 
but at the potential cost of validity. Our results show that training did increase 
the percentage of agreement among individual raters but it was not significant. 

This simulated anthropological study was done to estimate the 
accuracies and consistencies between individual and group perceptions on a 
relatively "neutral" culture (with fictitious cartoon characters). As indicated 
earlier, we wanted to determine the relative consistency between individuals v. 
groups, and we found that group predictions were not more consistent than 
individual predictions. We initially expected that it would increase consistency 
(replicability) when doing ethnographic research to have more than one 
observer looking at the same data. These studies, however, did not support this 
initial assumption. This research suggests that triangulation in the collection 
and analysis of data may be mote problematic than we initially expected. 

Our research, we believe, supports the need to do further research on 
qualitative methodology for the purpose of improving appropriate interpretation 
of qualitative research. The finding that our subjects thought they highly agreed 
with the group consensus when their responses indicated they did not has 
important implications for the interpretation of qualitative research.^ 



4Note: This investigation was done in three parts, the first two studies having 
been reported individually. Prior to doing Study 3, it was thought desirable to 
re-evaluate the categorizations and calculations arrived at in Study 2. What we 
found in doing this were some minor errors in the categorization process as well 
as a minor addition error. However, the corrections did not change the 
substantive interpretation of the data at all. . 
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*1 Question #3 Is: Describe this culture (what are the cultural values?). 
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•1 Question #3 is: Describe this culture (what are the cultural values?). 
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1. Eiljoy life/pleasure 

2. Anti-establishment, anti- 
mainstream, anti-government 

3. Pro-establishment, anti- 
mainstream, anti-government 

4. Speak for minorities 
(minority viewpoints, not 
just ethnic minority) 

5. Sexism (male and female) 

6. Importance of relationships 

7 . Xenophobic/labeling 
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9. Political correctness 
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11. Gender differences 
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15. Lack of existential factors 
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5. Importance of relationships 

6 . Xenophobic/labeling 
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8. Political correctness 

9. Altruism/personal growth 

10. Gender differences 

11. Effects of the media 
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